(19) 




Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 




(j cation: 

07.1 1 .200); Bulletin 2001/45 

(21) Application number 01105220.6 

(22) Date of filing: 20.03.1995 



(H) EP 1 152 589 A2 

EUROPEAN PATENT APPLICATION 

(51) mtci7: H04N 1/21, H04N 5/232 



(84) Designated Contracting States: 


• Fredlund, John R. 


DE FR GB 


Rochester, New York 14650-2201 (US) 




• Parulski, Kenneth A. 


(30) Priority: 29.03.1994 US 219608 


Rochester, New York 1 4650-2201 (US) 


(62) Document number(s) of the earlier appltcation(s) in 


(74) Representative: 


accordance with Art. 76 EPC: 


Lewandowsky, Klaus, Dipl.-lng. et al 


95104047.6/0 675 648 


Kodak Aktiengesellschaft, 




Patentabteilung 


(71) Applicant: EASTMAN KODAK COMPANY 


70323 Stuttgart (DE) 


Rochester, New York 14650-2201 (US) 






Remarks: 


(72) Inventors: 


This application was filed on 03 - 03 - 2001 as a 


• Sarbadhikari, Kamal K. 


divisional application to the application mentioned 


Rochester, New York 14650-2201 (US) 


under INID code 62. 



(54) Electronic imaging system using a removable software-enhanced storage device 



(57) A method for capturing and processing images 
in a digital camera (1) having a removable storage de- 
vice (24) for storing the images captured by the camera , 
comprising the steps of: 

(a) capturing and digitizing an image in response to 
a user input; 

(b) storing the captured image in an image buffer 
(1 8) capable of storing a plurality of images; 

(c) beginning the processing of the stored image us- 
ing a programmable processor (20); 



(d) if there is another user input during the process- 
ing step (c), interrupting the processing of the stored 
image in response to said user input, and repeating 
steps (a) and (b) for a second image; 

(e) continuing the processing of the first stored im- 
age after the completion of step (d); and 

(f) storing the processed image in the storage de- 
vice (24) and continuing with step (c) until all images 
stored in the image buffer (18) have been proc- 
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Description 
Field of Invention 

[0001 ] This invention pertains to the field of electronic 5 
imaging and, in particular, to electronic imaging with an 
electronic still camera that utilizes a removable storage 
device for storing images. 

BACKGROUND OF THE INVENTION 10 

[0002] As understood in the prior art, a digital elec- 
tronic stiil camera is a device which uses an electronic 
sensor to capture an image; signal processing to modify 
the captured image signal, e.g., as a function of ambient 
light, and to represent it numerically; and some storage 
device to preserve the numerical image data. Ordinarily, ' 
the image storage device is merely a receptacle for the 
image data. 

[0003] It is further known for a digital electronic still 20 
camera to use a removable storage device, such as an 
integrated circuit memory card, to store images. 
For instance, U.S. Patent No. 5,016,107 describes an 
electronic still camera utilizing image compression and 
providing digital storage in a removable memory card 25 
having a static random access memory. Software that 
controls the camera operation is located in non-remov- 
able programmable read only memory (PROM) integrat- 
ed circuits in the camera. In this camera, the integrated 
circuits in the removable memory card store image data 30 
and a directory locating the data. 
[0004] Other electronic camera systems described in 
the prior art record an image header along with the dig- 
ital image data. The header typically describes some 
characteristics about the image or the camera. For in- 35 
stance, in U.S. Patent No. 5,018,017, such header in- 
formation indicates the classification of the information; 
the imaging system in use; the date and time; the com- 
pression mode; as well as image-specific information, 
such as flash use, white balance data, exposure value, 40 
and shutter speed. Furthermore, U.S. Patent No. 
5,1 53,729 shows memory capacity, a free area pointer, 
a battery expiration date, and a write inhibit flag in the 
header area of a memory card. Typically, each system 
conveys such characteristic data from the electronic 45 
camera by way of the memory card (where the data is 
temporarily stored) to a peripheral device, such as a 
player. The object is to increase the downstream func- 
tionality of the system by allowing utilization of image- 
related data in subsequent processing of the image data so 
in a downstream peripheral device. The camera or im- 
aging device generally utilizes little of the stored image 
characteristic data, one exception being data regarding 
the utilization of the storage de vice itself , i fi^ftow much 
memory spac e is remaining for further^icture s. 55 
Irtthe-fatte Tcase, Uiu camera accesses sucn data to dis- 
play unlizanon to the camera user. " 
[0005] Coordination between a camera and a compu- 



ter is generally shown in US-A-5,475,441 , "Electronic 
Camera with Memory Card Interface to a Computer", 
which was filed on December 1 0, 1992 and assigned to 
common assignee with the present application. This ap- 
plication describes a camera that is specially shaped so 
as to plug directly into the memory card slot of a portable 
computer; the camera is then usable as a unit with the 
portable computer. The camera contains EPROM mem- 
ory with code for in-camera white balance and gamma 
correction, a nd also stores the compiled codej MhicbJs-^ 
used (by the computer) t o operate the camera and to 
proce ss the images from the sensor color filter arra yjo 
obt ain a full resolution, colo r corrected image^jliejn-i^' 
age data is captured^by^e.cameraand directfydown- 
loaded to the computer, where it is processed with code 



obtaii 



I from the camera. As with] 



' card 



vnstream (in the compu- 
terjto process image data. 

[0006] In a camera intended for acquisition of image 
data for use in a computer, such as described in US-A- 
5,475,441 , it is well understood that data corresponding 
to captured images may also be stored in a storage de- 
vice that is capable of beingj^nFKwedTf^^ 
Such a storage devjcelsoften considered an alogoft&to 
film in a conventional film camera. Just as it is known Jto 
encode fjJrn with marks that are useful in downstre 

ang of the film, it is also known for an electronic 
imacfe storage device to contain non-image data which 
is liseful in modifying the image data. Following the film 
analogy, such additional data is usjefdfin downstream 
prc^s^ir^suchasjnji£om to modify the image 
data. Unlike film, however, the additional non -image da- 
ta may be read by a variety of peripheral devices, in- 
cluding the camera itself. This provides an opportunity 
to deal with a number of imaging problems. 
[0007] For instance, in most known electronic imaging 
systems, the software which controls the camera ordi- 
narily cannot be updated without purchasing an entirely 
new camera, containing a lens, CCD image sensor, etc. 
(Albeit, a few high end film cameras utilize a removable 
program module from which the camera can download 
exposure algorithms.) It is well known that computer 
programs are updated more regularly than computer 
hardware, since the software can be refined to provide 
higher performance (i.e. better image quajityXot-^ore 
features while using the same haixhy ateT ^nelectro nte 
racan use arprogrammable digifaTpTOcessorcorP" 
ailed by software Such q camera offers an oppogfrmfty- 
for software updates, if the system is designed to facil- 
itate such updates. What is needed Is a method for soft- 
ware updates of the camera algorithms for higher per- 
formance, and for adding new features to a camera. 
Such a method should be quick and convenient for the 
user, preferably without requiring the use of additional 
storage modules. Existingmempjy_device^ 
of limited two^ay-^xmirnllnlcation with an electronic 
camer^^-stJchi as to report to the camera on memory 
availability. An opportunity exists for en hap 
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such "upstream" communication to provide altogether 
new applications for an electronic camera. 

SUMMARY OF THE INVENTION 

[0008] The opportunity in the prior art for upstream uti- 
lization of the memory capability of a removable storage 
device is embodied, according to the invention in claim 
1, by an electronic imaging system including an elec- 
tronicj^amera for capturing and sto ring images irL a-re- 
jTf6vabie~^rag&^ with 
(software forj>p_ejratiMthe imaging^ system. Tha i 
system includes an opfic^t^ection for establishing the 
optical parameters of image capture, an image sensing 
section for electrically capturing an image provided by 
the optical section, a signal processing sectionjor 
proce ssing the electrically captured image<^progranrK > 
a^liprocessor for c ontrolling the sectior fe-oHhe'S^^ 
- - J — ~ — foTToading the programmaoie procei^ 
Dpwitn software preloaded in the removable storage 
device for affecting operation of one or more of the sec- 
tions of the system. 



5quentiy, the inventio n pn 
tronjcsMpholograefcy-s^^ 

software located on a removable memory (e.g., a v 
memory card) used with an electronic still 

use the remo vable memofyttrsfore im- 
ages and data related to an individual image (date, color 
temperature of the iiluminant, etc.) or data related to the 
camera (type of color filter array, sensor defect loca- 
tions, etc.) which is transferred to an image reproducing 
device. Except for data about memory space, the com- 
munication is one way. A camera according to this in- 
vention, however, uses the removable media to transfer 
software previously recorded on the memory card (such 
as improved camera firmware to process the image, or 
a special access code) to the camera. Th e advanta- 
geous effect is that t he operation of the camera or play- 
bacfraevice can be im proved some years after the cam- 
ei a-ferpm chased , by using an "enhanced" m emory card 
^ oritairfihlTimproved software for o perating the camera 
or playback device. Moreover, such two-way communi- 
cation opens use~oTthe storage device to other en- 
hancement possibilities, e.g., to modify image data in 
selected ways, to modify camera performance for spe- 
cial situations, or to communicate non-captured images 
(special overlays) to the camera. 



process enhancement files; 
Figure 3 is a flow diagram of the steps involved in 
applying an enhancement algorithm to the camera 
of Figure 2; 

5 Figure 4 is a block diagram of the capture chain 
showing the points at which processing according 
to the invention may occur; 
Figure 5 is a flow diagram of the steps involved in 
processing an image sometime after storage in the 
10 camera of Figure 2; 

Figure 6 is a flow diagram of the steps involved in 
interrupt processing in the camera of Figure 2; 
Figure 7 is a flow diagram of the steps involved in 
controlling access to processing in the camera of 
15 Figure 2; 

Figure 8 is a pictorial illustration of the application 
of pre-existing overlay images to images captured 
by the camera of Figure 2; 
Figure 9 is a pictorial illustration of the use of an 
electronic viewfinder in the camera of Figure 2 to 
frame the captured image with respect to an overlay 
image; 

Figure 10 shows an additional embodiment of an 
electronic camera configured according to the in- 
vention; and 

Figure 11 shows an electronic imaging system in- 
corporating the camera of Figure 1 0 by means of a 
cable connection. 

30 DESCRIPTION OF THE PREFERRED 



35 



40 
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50 



BRIEF DESCRIPTION OF THE DRAWINGS 

[0010] The prior art and the invention will be described 
in relation to the drawings, wherein 



Figure 1 is a diagram of a known electronic imaging 
system incorporating an electronic camera for in- 55 
putting images to a computer; 
Figure 2 is a block diagram of an electronic camera 
configured according to the invention to receive and 



EMBODIMENTS 

[0011] Because electronic still cameras employing 
electronic sensors are well known, the present descrip- 
tion will be directed in particularto elements forming part 
of, or cooperating more directly with, apparatus in ac- 
cordance with the present invention. Elements not spe- 
cifically shown or described herein may be selected 
from those known in the art. 

[0012] Figure 1 shows a known electronic still photog- 
raphy system useful for inputting images to a computer. 
The system includes an electronic camera 1 with an 
electronic sensing section la, a digital processing sec- 
tion 1 b, a memory card slot 2, a removable memory card 
3, a computer 4 with a built-in memory card reader 5, 
and a printer 6. Images are captured by the sensing sec- 
tion la in the camera 1 , manipulated in the processing 
section 1b, stored on the memojy^^aiit-SrtmaV^ns- 
ferred from the card^o~Tfie~computer 4. The sol 
use^Uo'Control the sensing and processing sections la 
'Und lb in the camera is entirely located in firmware.mi 
ory 7 in the camera landj^uwor-bTTF^ without 
?ra cover, unsoldering the firmware 
memory circuit, and replacing it with a new circuit con- 
taining updated code. The application software used to 
control the playback operations of the computer is sup- 
plied, at least in part, by the camera manufacturer. It is 
stored in a computer hard drive 8, and downloaded to a 
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computer random a^e^,memoiy4BAM)jTierTi ory 9a, 
where thepr^ranlTsusetfby the central proce§smp^un It 
(CPy>9tr1b update this software, the user must be pro\ 
[led with a new program on a floppy disc, which can; 
' be transferred to the hard drive 8. For exarnple j jhejiep'- 
r may ™ntajn « "piug4^-ffKwiDte~f^^ 

The 
be 



software application provided with the camera, 
plug-in contains both code to allow the images^ 
downloaded from the camera to thecomputer, and"spft- 
ware code to create color image s from the di gjtge^cam- 
v era image-dataT I he sonwarecbde implements algo- 
rithms for processing the color filter array sampled im- 
age data supplied by the camera. These algorithms in- 
clude color interpolation, color balance, and color cor- 
rection. A typical color interpolation algorithm is de- '5 
scribed in greater detail in US-A-5, 373, 322, "Apparatus 
and method for adaptivefy interpolating a full color im- 
age utilizing chrominance gradients", filed June 30, 
1993, which is assigned to Eastman Kodak Company. 
[0013] The invention is based on the recognition that 20 
the storage device, i.e., the memory card 3 in Figure 1 , 
need not be primarily oriented to storage of data for 
downstream (i.e., computer) processing. The storage 
device may additionally contain files (data, code, etc.) 
which are capable of u^tream'modifTcation-ofJm^e 25 
data arjd camera performance. This provides an oppor-^ 

r for affectinjpijap^-data-<^^ 
QajtipjilatioTTstages prior to storage. Furthermore, 
these files may be accessed automatically by the cam- 
era, or selected by the user by means of appropriate 30 
intervention through the camera. Therefore, the remov- 
able image storage device is capable of two-way com- 
munication with the camera. That is, the card slot inter- 
face is not limited to "dumping" image data downstream 
to the removable storage device, but also allows com- 
munication from the removable storage device up- 
stream to the camera as well. Inasmuch as the files are 
software which enhance the operation of the camera, 
the digital storage device may be viewed as a software- 
enhanced device, and the files are hereinafter referred *o 
to as enhancement data files. 

[0014] A number of advantages arise from such two- 
way communication. Enhancement files can be includ- 
ed in the digital storage device whicfi are not in them- 
selves images. ImagejMpceestntJ sottwarejook-up la- 45 
bles, matrices, compression tables: dynamic range op- 
timizatiol ftaples TanSother filesxapable-of^eetingthe 
captured* image data can be included in the digital stor- 
age device. Typical algorithms which can be modified, 
or updated, thus include color filter array interpolation so 
algorithms, noise reduction algorithms, edge sharpen- 
ing algorithms, color reproduction algorithms, compres- 
sion algorithms, or dynamic range optimization algo- 
rithms. In this manner, the camera need only be capable 
of executing the functions and transformations con- ss 
tained in the storage device, and need not require the 
storage capacity to contain all information necessary to^^ 
execute the processing of the image data. Such no f n^-_ 



image files, which cause processing of the captured im- 
age data, may process the captured image data such 
that the most "true-to-scene" reproduction is achieved, 
or instead provide speci al^ effects of the kind which 
change the imagezinto'Bome, desired altera!] 
originaiscetfeT 

1*5] Another type of non-image file which can be 
contained in the digital storage device enhances the op- 
eration of the camera. Enhancement files can be includ- 

I which control capture parameters such as expos 
timeTapeTtaro ootting, flash ranpp niitj*rtraTlo r soon. A 
particular set of control parameters can be used to tailor 
the capture parameters to the type of scene or subject 
being captured. This type of file might also allow diag- 
nostic tests of the camera to insure proper operation and 
to detect fault conditions in the camera system. A further 
type of enhancement file contained in the storage device 
comprises pre-existing image data files, i.e., files with 
images not captured by the camera system. Such files 
are accompanied by code which instructs the camera 
on how to merge the pre-existing image files with those 
captured by the camera. In this manner graphic overlays 
and text can be superimposed on the captured images 
bythe camera. Proper execution of such afeature might 
include a user interface for selection of pre-existing im- 
age and overlay combinations, and also a viewfinder 
which helps the user to compose the captured image so 
as to complement the selected overlay. 
Whether for image processing, enhanced performance, 
or pre-existing images, the algorithms themselves are 
jconventional and not themselves part of this invention. 
However, where they are stored, and how they are used, 
is within the scope of the invention. In that connection 
^another function of the invention is to store replacement, 
or updated, software files in the digital storage device 
so that software updates can be added to the camera 
after its initial purchase by the subsequent purchase of 
appropriately-enhanced storage devices. 
[0016] The invention also concerns the sequence of 
execution of operations made possible by files con- 
tained in the image storage device. It is anticipated that 
the capture of images may occur in a rapid and/or ran- 
dom fashion. It may be undesirable to interrupt image 
processing to initiate another capture. If that is the case, 
processing above and beyond that which is necessary 
to achieve initial storage would not have to occur until 
the camera is no longer in a "ready" mode. That is, no 
processing takes place until the camera is no longer be- 
ing used to capture images and is placed in an "off" or 
"sleep" mode. Alternatively, the camera could interrupt 
processing operations to carry out capture operations. 
Once the capture is complete, the camera can return to 

processing operations. ^ - — — ^ 

[0017] Figure 2.shows an electronic still camera ca- 
pabteof-using a "software enhanced" storage deviceas^ 
-taught by the invention. The electronic stHljamerais 
divided generally into an inputsecliofrA^ndacompres- 
_sjpji_andj£coFdifl§-sB^^ input section A in- 
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eludes an exposure section 10 for directing image light 
from a subject (not shown) toward an image sensor 12. 
A flash unit 11 may be used to illuminate the subject 
Although not shown, the exposure section 10 includes 
conventional optics for directing the image light through 
a diaphragm, which regulates the optical aperture, and 
a shutter, which regulates exposure time. The sensor 
12, which includes a two-dimensional array of pho- 
tosites corresponding to picture elements of the image, 
is a conventional charge-coupled device (CCD) using, 
e.g., either well-known interline transfer or frame trans- 
fer techniques. The sensor 1 2 is exposed to image light 
so that analog image charge information is generated 
in respective photosites. The charge information is ap- 
plied to an output diode 14, which converts the charge 
information to analog image signals corresponding to 
respective picture elements. The analog image signals 
are applied to an A/D converter 1 6, which generates a 
digital image signal from the analog input signals for 
each picture element. 

[001 8] The digital signals are applied to an image buff- 
er 18 for storing one or more images, albeit shown in 
Figure 2 as a random access memory (RAM) with stor- 
age capacity for a plurality of still images. A programmed 
control processor 20 generally controls the input section 
A of the camera by initiating and controlling exposure 
(by operation of the diaphragm and shutter (not shown) 
in the exposure section 1 0), by controlling the flash unit 
11, by generating the horizontal and vertical clocks 
needed for driving the sensor 1 2 and for clocking image 
information therefrom, and by enabling the A/D convert- 
er 16 in conjunction with the image buffer 18 for each 
signal segment relating to a picture element. (The con- 
trol processor 20 would ordinarily include a microproc- 
essor coupled with a system timing circuit.) Exposure 
control would typically be effected by using ambient light 
information from a photocell 1 9. 
[001 9] Once a certain number of digital image signals 
have been accumulated in the image buffer 18, the 
stored signals are applied to a programmed digital sig- 
nal processor 22, which controls the throughput 
processing rate for the compression and recording sec- 
tion B of the camera. The algorithms and other operating 
code used by the processor 22 are stored in the algo- 
rithm memory 28. The digital signal processor 22 com- 
presses each still image stored in the jmage buffer 18 
according to^ajenown Image compressioXalgorithm, 
such as .the well-known JPEG (Joint Photographic Ex- 
perts Group) discreje_^osine-4ransforniation-based 
^mpression^algohthm. The processor 22 applies a 
compression algorithm from the memory 28 to the digital 
image signals, and sends the compressed signals to a 
removable storage device via an interface 26. While a 
memory card 24 is shown in this system as the storage 
device, other devices may be used, such as a floppy 
disk magnetic medium, a small hard drive, or optical 
storage (in the latter cases, suitable conventional read- 
ing/writing apparatus would be provided in the camera, 



e.g., magnetic or optical read/write head, etc.) A repre- 
sentative memory card is a card adapted to the PCMCIA 
card interface standard, such as described in the PC 
Card Standard, Release 2.0, published by The Personal 
5 Computer Memory Card International Association, Sun- 
nyvale, California, September 1991. Moreover, while 
compression is shown in the camera of Figure 2, com- 
pression is not a necessary feature of the invention, and 
uncompressed data could be stored instead. 

10 [0020] The memory card 24 contains solid state mem- 
ory 24a, such as Flash EPROM memory, which the card 
uses to store image data files. In addition, the memory 
card 24 contains additional memory to store enhance- 
ment data files 24b to modify the camera operation, or 

is the images captured by the camera, or to provide pre- 
existing overlay images. These software enhancements 
are normally programmed by the card manufacturer be- 
fore the card is sold to the user, and can be stored in the 
Card Information Structure (CIS) of the PCMCIA Format 

20 (as described in the aforementioned Release 2.0). The 
■software enhancement" code may be stored in sepa- 
rate non-erasable memory on the card 24, or in a portion 
of the memory space of the memory 24a used to store 
the image data. 

25 [0021] The input section A operates at a rate com- 
mensurate with normal operation of the camera while 
compression, which may consume more time, can be 
relatively divorced from the input rate. The exposure 
section 10 exposes the sensor 12 to image light for a 

30 time period dependent upon exposure requirements, for 
example, a time period between 1/1000 second and 
several seconds. The image charge is then swept from 
the photosites in the sensor 12, converted to a digital 
format, and written into the image buffer 18. The repe- 

35 tition rate of the driving signals provided by the control 
processor 20 to the sensor 1 2, the A/D converter 1 6 and 
the buffer 1 8 are accordingly generated to achieve such 
a transfer. The processing throughput rate of the com- 
pression and recording section B may be determined by 

*o the character of an image, i.e., the amount of detail ver- 
sus redundant information, and the speed of the digital 
signal processor 22. 

[0022] An operation display panel 30 is connected to 
the control processor 20 for displaying information use- 

45 ful in operation of the camera. Such information might 
include typical photographic data, such as shutter 
speed, aperture, exposure bias, color balance (auto, 
tungsten, fluorescent, daylight), high/low resolution, low 
battery, low light, exposure modes (aperture preferred, 

so shutter preferred), and so on. Moreover, other informa- 
tion unique to this type of camera is displayed. For in- 
stance, the memory card 24 would ordinarily include a 
directory signifying the beginning and ending of each 
stored image. This would show on the display 30 as ei- 

55 ther (or both) the number of images stored or the 
number of image spaces remaining, or estimated to be 
remaining. In certain applications, an electronic view- 
finder 29 would be included for displaying images either 
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before or after storage. 

[0023] The control processor 20 also generates im- 
portant information about the condition and capability of 
the memoiycfl c d 24. S pecific a lly , the i nter! 

resence of a card 24 and, if no card is con- 
foduced on the"< 
■ acorc Us-pi 





a "card full" display is produced. While not 
shown, the display 30 may show the number of "remain- 
ogj mages. " If enhancementfilesj>4b_a^ 
thecara 24, tney can oe idenrffiedon the operation dis- 
play 30. For instance, performance enhancements, 
such as special set-up for portraiture, can be identified; 
processing enhancements, such as special effects, can 
be identified; and overlay enhancements, such as birth- 
day or holiday picture borders, can be identified. User 
intervention in order to specify the several enhancement 
opportunities, or to set conventional capture parame- 
ters, is obtained from a user inputs section 21 . Such in- 
puts could be one or more selector sw itches op erable 
alone or in corijujctjpiLwith prompts fjggvthe < 
r the electronic viewfinder 29. 
24] The camera also contains firmware memory 
32 with operating code-to r the camera, as well as RAM 
instruction memory 31 connected to receive code frorn^ 

tho mamnruroH Whop mp mPmOrYH 



; ope rati 



i in- 



to a single file. Anyj»mbination of imagej 
files may be used. 
[0025] After the images from the camera are stored 
on the memory card 24, the card is inserted into the com- 
puter memory card reader 5 (see Figure 1). In certain 
applications where the memory card may contain "soft- 
ware enhancements" which can be used by the compu- 
ter, the computer uses the software supplied with the 
camera to check if the memory card contains such al- 
gorithms. If not, the software algorithms supplied with 
the camera (and stored in the drive 8) are downloaded 
to the computer RAM memory 9a used by the CPU 9b 
to process the image. If the card does contain appropri- 
ate "software enhancements", however, these are 
downloaded from the card to the RAM instruction mem- 
ory 9a and used by the CPU 9b in place of some portion 
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20 



serted mto the camera, the processor 20 uses the 
firmware algorithms in the memory 32 to determine if 
thememory card 24 contains "software enhancements" 

fiich can be used by the camera. If not, the firmware 30 
algorithms supplied with the camera in the memory 3£ 
are used by the programmable processor 20. If the can 
does contain appropriate "software enhancements' 
however, these are downloaded from the enhancemen 
file section 24b of the card 24 to the RAM instruction 35 
memory 31 and used by the programmable process 
20 in place of some portion of the firmware algorithi 
supplied with the camera from the firmware memory 
this way the camera can execute new, improved/aJ- 

ial 40 

features^More than one algorithm file may alsk be 
downloaded^or different functions maybe 1 ■ 
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of the software algorithms from the hard drive 8. This 
allows a quick and convenient way of updating the al- 
gorithms used by the computer, without requiring a spe- 
cial update disk to be distributed to users. These image 
processing algorithms can be applied by the camera ei- 
ther during initial processing, when the image is first 
captured, processed, and stored, or post-capture. 

026] According to the invention shown in Figure 2, 
the images are transferred from the camera to the com- 
puter via the removable memory, rather than a cable in- 
terface. Therefore, the camera does not have to be in 
the same location as the computer. This makes it pos- 
sible to view the images from the camera in many dif- 
ferent locations, on many different computers, as long 
as the computer has the ability to both accept the re- 
movable memory and correctly process the image data. 
Since the removable memory can be used to store code 
for processing the image data in the computer, this code 
can include the color interpolation, color balance, and 
correction algorithms described in connection with Fig- 
ure 1 , or future improved versions of these algorithms. 
The advantage of storing the algorithms on the remov- 
le memory is that it eliminates the need for a separate 
eans of supplying the code, such as the floppy disk. 
Supplying the algorithms along with the images on the 
removable memory, such as a PCMCIA card, makes it 
possible for any computer capable of reading the image 
data from the card to also download and utilize the al- 
gorithms needed to process the image data in order to 
create a full color image. This makes it convenient to 
view the images on almost any computer which includes 
a PCMCIA card connector, without requiring that the 
computer be supplied with a special plug-in to provide 
the image processing code, such as for a color filter ar- 
ray interpolation. 

[0027] Apart from updating camera algorithms, the 
enhanced algorithm files might be tailored in a specific 
manner for memory cards sold for particular applica- 
tions. In electronic still imaging systems, the resolution 
and/or photographic speed of the camera may be al- 
tered by, e.g., subsampling and/or electrical gain adjust- 
ments. A type of non-image data file providing such ad- 
justments would be included in image data storage de- 
vices which are designed for specific capture applica- 
tions. In the same manner that certain film speeds are 
selected for films, an image storage device might be se- 
lected for action photography, portraiture, or infant pho- 
tography. An appropriate set of parameters for image 
capture would be incorporated in the associated non- 
image data file. One possible application is portraits. In 
this application, the non-image data file would provide 
that the color reproduction should be optimized for skin 
tones, the edge sharpening should not be too severe, 
and the color filter array algorithm should minimize color 
aliasing at the expense of reduced sharpness. Other ap- 
plications could also be envisioned. 
[0028] The steps involved in applying an image 
processing algorithm are shown in the flow diagram of 



6 



11 



EP 1 152 589 A2 



12 



Figure 3. When the camera is ready to capture images, 
the processor 20 within the camera checks if there are 
software enhancements in the enhancement file 24b in 
the card 24. If such files are present, the processor 20 
downloads the processing algorithm(s) in the enhance- 
ment file 24b contained in the removable memory card 
24. If there is more than one algorithm, the user would 
preselect the appropriate one from the user input sec- 
tion 21 , or the camera would select one according to the 
detected picture taking situation; absent any selection, 
the camera could use a default (or no) algorithm. Once 
the exposure is made, processing may commence. Fig- 
ure 4 is a block diagram of the capture chain showing 
several points at which processing may occur. The al- 
gorithms may be applied at point 40 after the image data 
is digitized. Processing according to the downloaded al- 
gorithm may alternatively take place as the image data 
is originally being processed for initial storage at points 
42 or 44, or may be applied at a later time after the initial 
storage of the image is complete at point 46. Figure 5 is 
a flow diagram showing the steps of subsequent 
processing at point 46 (Figure 4), including after a time 
lapse. There may be utility in storing the initial capture 
in raw form so that different algorithms can be applied 
to achieve the most desirable result. The processing al- 
gorithms themselves, examples of which were de- 
scribed earlier, are conventional and not themselves 
part of this invention. 

[0029] For all image data manipulations, it may be ad- 
vantageous to postpone image data processing until 
sometime after the capture is made as generally implied 
by processing point 46 in Figure 4. This is to say that 
any processing which is done to the image beyond that 
which is necessary for initial storage may occur when 
the camera is in the "sleep" or "off" mode. This may be 
desirable so that the camera can be ready for the next 
exposure in the minimum time possible, and not be bur- 
dened with post-processing of the image during a time 
when image capture may still occur. 
[0030] An alternative to delayed process in g is to com- 
mence processing as soon as the capture is complete, 
and to interrupt the processing when the user indicates 
another capture must take place, as shown in Figure 6. 
This is an acceptable means of processing the images, 
but may become complex if a number of images are cap- 
tured in rapid succession. Delayed processing, as 
shown in Figure 5, may be preferred. A flag in the image 
file header may be set to indicate if the image has been 
processed or not. A camera in the midst of a processing 
operation may be programmed to return to "ready" 
mode only after all processing for the current image is 
complete. 

[0031] In addition to storing complete software algo- 
rithms, the software enhancement located in the en- 
hancement file 24b on the card 24 can be codes which 
are used to allow access to certain features of the soft- 
ware already located in the camera (or computer), or to 
select among specific options for the software located 



in the camera (or computer). An example of the use of 
such access codes is shown in the flowchart of Figure 
7. When the card 24 is inserted into the camera 1 , the 
camera checks to determine if the card is a normal mem- 

5 ory card, or a "software enhanced" card. For PCMCIA 
cards, the use of the "software enhanced" card might be 
indicated, for example, by the use of a digital encryption 
key as part of the Card Information Structure. The card 
also includes special access codes for exposure index, 

10 color/monochrome, color reproduction and portrait. The 
camera may include variable gain corresponding, e.g., 
to ISO exposure indexes of 50, 100, 200, and 400. For 
normal cards, the camera speed is set at ISO 100. For 
software enhanced cards, the camera checks to see 

is which speed should be used for the photos, depending 
on which version of the card the user has inserted into 
the camera. In this embodiment, the camera includes 
algorithms in the firmware memory 32 necessary for var- 
iable gain, and for storage of either color images or the 

20 monochrome record from the color image sensor 12. 
For software enhanced cards, a "black and white" bit of 
the card is checked, to see which algorithm should be 
used. The user then takes the desired photos, and in- 
serts the card into the computer to print the images in 

25 the printer 6. When the card 24 is inserted into the com- 
puter 4, the computer checks the Card Information 
Structure to determine if the card is a normal memory 
card, or a "software enhanced" card. For normal cards, 
the normal processing is used. For "software enhanced" 

30 cards, more elaborate color reproduction algorithms 
might be used, to provide higher quality images. The 
card's "portrait bit" is also checked, to see if the user has 
selected a card for taking portraits. If so, the special 
processing is used to soften the image and reduce noise 

35 which might appear as facial blemishes. The image is 
then printed by the printer. 

[0032] "Pre-exposed" image data files can also be 
resident in the enhancement file 24b in the removable 
image data storage device. These files would include 

40 image templates or overlays for combination with user- 
captured images, as shown pictorially in Figure 8. These 
files would likely be computer generated surrounds hav- 
ing the purpose of enhancing the images captured by 
the user for particular situations. The manner in which 

45 these files are combined with user captured images can 
be automatic or with user intervention through the user 
input section 21 . In the camera, the processor 20 would 
retrieve the overlay and the digital processor 22 would 
insert the user captured image into the overlay sur- 

so round. A given removable image storage device might 
be labeled as a Christmas Album, for example, and the 
camera would insert the user captured images into the 
seasonal templates without input from the user. The 
camera may prompt the user to frame the image appro- 

55 priatefy with cues in the electronic viewfinder 29, as 
shown in Figure 9. An outline of the template might ap- 
pear in the viewfinder 29 to assist with framing. The 
camera should be capable of understanding where the 
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"pre-exposed" overlay wilt appear in the final combined 
image in order to assist the user with framing. A file con- 
veying this information may accompany the overlay data 
file, the information may be contained in the overlay data 
file header, or the camera may be capable of interpreting 
the location of the overlay by reading the overlay data 
file itself. Many different types of overlays may be used. 
The user could select removable image storage devices 
expressly for different seasonal or special events such 
as birthdays or sporting championships. 
[0033] Implementation of graphic overlays can also 
be accomplished with user input. The user would select 
images and overlays according to personal preference 
after observing them in the viewf inder 29 or on a views- 
creen (not shown) included in the camera. This ability 
implies an ability to browse both images and overlays 
contained on the image storage device by means of the 
camera. The camera must be capable of reading and 
displaying images from the removable storage device, 
and allowing the user to select both independently. In 
certain applications, more than one captured image 
could be inserted into an overlay. Alternatively, while a 
single image may be inserted, the framing of the image 
in the viewfinder relative to the overlay need not be in 
one-to-one correspondence between the captured im- 
age and the displayed image. That is, the captured im- 
age may be reduced so that the whole captured image 
is visible in the display through the overlay. Likewise, 
the captured image could be rotated or otherwise trans- 
formed as to position before being inserted into the over- 
lay. Moreover, while the captured image may be shown 
through the overlay in the viewfinder, the camera may 
not actually combine these images, but rather create a 
script file which would direct the computer to do the 
proper combination. With adequate internal memory in 
the buffer 18 or multiple storage inputs (e.g., card slots 
26), the camera might also allow selection of an overlay 
from one removable storage device to be combined with 
a captured image resident on another. 
[0034] While the invention has been described with 
particular reference to several embodiments, it will be 
understood by those skilled in the art that various chang- 
es may be made and equivalents may be substituted for 
elements of the embodiments without departing from in- 
vention. For example, instead of using a removable 
memory card 24, the image data files 24a and an nance- 
ment files 24b could be stored in non-removable elec- 
trically programmable non-volatile memory located in- 
side the camera. As shown in Figure 11, image data 
would be downloaded from the non-volatile image mem- 
ory 35 i n the camera 1 to the computer 4 over a standard 
electrical computer interface cable 38, such as a RS- 
232 or SCSI interface connection. The enhancement file 
would be supplied on a floppy disk, downloaded to the 
computer 4, and then uploaded by the computer 4 to an 
electrically erasable firmware memory 36 in the camera 
1 via the same interface cable 38. This additional em- 
bodiment of the invention is further shown in Figure 10 



in relation to an interface section 34 through which im- 
ages are transferred to the computer 4 and operating 
software is uploaded from the computer to the electri- 
cally erasable firmware memory 36 in the camera. Fig- 

5 ure 11 generally shows the connection of the camera of 
Figure 1 0 to a computer 4 by means of a standard elec- 
trical computer interface cable 38. In addition, many 
modifications may be made to adapt a particular situa- 
tion without departing from the essential teachings of the 

10 present invention. It is accordingly intended that the 
claims shall cover all such modifications and applica- 
tions as do not depart from the scope of the invention 
as claimed. 
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Claims 



1 . A method for capturing and processing images in a 
digital camera (1 ) having a removable storage de- 
20 vice (24) for storing the images captured by the 
camera , comprising the steps of: 



(a) capturing and digitizing an image in re- 
sponse to a user input; 

(b) storing the captured image in an image buff- 
er (18) capable of storing a plurality of images; 

(c) beginning the processing of the stored im- 
age using a programmable processor (20); 

(d) if there is another user input during the 
processing step (c), interrupting the processing 
of the stored image in response to said user in- 
put, and repeating steps (a) and (b) for a sec- 
ond image; 

(e) continuing the processing of the first stored 
image after the completion of step (d); and 

(f) storing the processed image in the storage 
device (24) and continuing with step (c) until all 
images stored IrTtRe^image^buffer (18) have 

^been processed. 

2. The method of Claim 1 wherein the interrupting stop 
) further comprises the step of: 

storing a partially processed image such that 
ontinued processing in step (c) is resumed where' 
thVjnteruption occurred. 
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